Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization

Averaging scheme has attracted extensive attention in deep learning as well traditional machine learning. It achieves theoretically optimal convergence and also improves the empirical model performance. However, there is still a lack of sufficient analysis for strongly convex optimization. Typically, about last iterate gradient descent methods, which referred to individual convergence, fails attain its optimality due existence logarithmic factor. In order remove this factor, we first develop averaging (GDA), general projection-based dual algorithm setting. We further present primal-dual cases (SC-PDA), where primal schemes are simultaneously utilized. prove that GDA yields rate terms output averaging, while SC-PDA derives convergence. Several experiments on SVMs models validate correctness theoretical effectiveness algorithms.